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Augmented Vision System Using Wireless Commimications 

FIELD OF THE INVENTION 

The present invention pertains to augmented vision systems, such as may be 
5 used in surveying and machine control applications. More particularly, the present 
invenhon relates to an augmented vision system which makes use of a wireless 
communications device to receive data for generating images. 
BACKGROUND OF THE INVENTION 

Traditional surveying involves two operators working with a theodolite and 
1 0 range pole, or a more complex optical electronic "total station". One operator generally 
•■3 positions the theodolite over a known control point while the other holds the range pole 

'Is? 

^0 at a series of known or unknown points whose positions are to be checked or measured. 
J A prism mounted on the range pole is sighted through the theodolite and accurate 
1^ angular and distance measurements to the prism are taken at each point. The positions 
% of the points can then be determined by trigonometry. 

p An analogous process takes place in modern satellite based surveying. Current 

''"'4 

\1 techniques involve a reference or base antenna /receiver located over a known point 

\'i and a single operator who moves about with a roving anterma/ receiver or "GPS total 
station". The operator stops on various generally unknown points to record position 

20 information in a data collector using signals transmitted by a minimum number of 

satellite sources which are above the horizon. Correction data is transmitted from the 
base site through a telemetry system. The roving antenna is also carried on a range pole 
which is held by the operator, although the anterma need not be within sight of the 
reference antenna. A vector or base line is determined from the reference site to the 

25 rover. 

In real time techniques, an actual position is determined and recorded at each 
point during a survey. Other techiuques require post-processing in which data from 
both the reference and roving receivers is recorded for analysis and determination of 
actual position coordinates later. Most techniques are also either differential or 
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kinematic. In kinematic surveying, at least four satellites must be in view of each 
antenna at all times and centimeter level accuracy can currently be obtained. Five 
satellites are required for initialization. Differential surveying allows satellites to be 
temporarily blocked by obstructions between measurement points, and can provide 
5 submeter accuracy, which is sufficient for many purposes. In both kinds of technique, 
actual positions are calculated as latitude, longitude and height with reference to the 
global ellipsoid WGS-84 or an alternative datum. Local northing, easting and elevation 
coordinates can then be deternnined by applying an appropriate datum transformation 
and map projection. 

The satellite positioning system most commonly in use today is the Global 
'2 Positioning System (GPS), although other systems such as the Global Orbiting 
O Navigation System (GLONASS) are also in use or under development. Some land 
•g based systems which simulate satellite systems over a small area are also being 
JS developed to use non satellite signal sources. GPS is based on a constellation of at least 
)3 24 satellites operated by the U.S. Department of Defense. The satellite positions are 
'4 monitored closely from earth and act as reference points, from which an 
ly antenna /receiver in the field is able to determine position information. By measuring 
^ the travel time of signals transmitted from a number of satellites, the receiver is able to 
determine corresponding distances from the satellites to the antenna phase center, and 
20 then the position of the antenna by trilateration. In the past the information content of 
the satellite signals has been deliberately downgraded for civilian users, creating the 
need to use a reference station for accurate work as mentioned above. 

Surveyors and other operators carrying out survey related work use a range of 
equipment and procedures as will be described further below. A surveyor in the field 
25 typically carries a survey control device which provides a portable computer interface 
to the antenna/receiver. The surveyor generally navigates around a site, setting out or 
checking the layout of survey points, and recording attribute information for existing 
features, using the control device as required. The device typically contains a database 
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of points on the site, recorded or estimated during earlier work, and offers a variety of 
software functions which assist in the survey procedures. The operator is able to input 
information and commands through a keypad on the device, and view position 
coordinate data, and numerical or graphical results of the software calculations on a 
5 small display. For example, when staking out an item such as a line, arc, slope or 

surface on the site, the item is defined using existing points, a design point is specified 
as required, and the surveyor navigates to the point under guidance by the control 
device. A stake is placed in the ground as closely as possible to the point, and the 
position of the stake is accurately measured using the range pole. 
1§ Under other circumstances, an operator carrying out survey related work may be 

'^5' involved on a construction site, such as a building or road construction project, setting 
Q out or checking survey points and design features as work progresses. For example, the 
'I4 operator may be a surveyor or engineer who guides construction workers to ensure that 

a design is completed according to plan. On other sites, workers such as machine 
)3 operators may be acting independently of a surveyor, following a simple plan based on 
'4 survey work carried out at an earlier date. For example, a worker operating an 
rd excavator may remove earth from a ditch in order to lay or repair a utility conduit 
along a surveyed path. Another worker operating pile driving equipment may place 
piles to create foundations for a building or wharf according to a grid of surveyed or 
20 calculated locations. 

In each case described above, the surveyor, engineer, or machine operator makes 
use of survey related information and visual observations of a physical environment 
while pursuing their work procedures. These individuals would benefit from 
technology which provides them with richer, more complete and up-to-date 
25 information for use in carrying out the above-mentioned operations. For example, it 
would be desirable to have survey related equipment which provides an operator with 
augmented vision capabilities while at a job site, so as to provide the operator with 
information and other visual cues that are not normally visible or available. As another 
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example, the Internet is a vast medium for both communication and storage of 
information of many types. It would be desirable to make use of the Internet's 
information storage and communication potential in surveying and other related 
operations. 
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SUMMARY OF THE INVENTION 

The present invention includes a system for facilitating survey operations, which 
includes a wireless hand-held communication device, a display processor, and a 
portable display device. The wireless hand-held communication device receives 
survey-related data from a remote processing system via a wireless network, and the 
display processor generates image data based on the survey-related data. The portable 
display device receives the image data from the display processor, and has a 
substantially transparent display area to superimpose an image on a field of view of a 
user based on the image data. 

Other features of the present invention will be apparent from the accompanying 
drawings and from the detailed description which follows. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is illustrated by way of example and not limitation in the 
figures of the accompanying drawings, in which like references indicate similar 
elements and in which: 
5 FIG. 1 schematically shows two survey operators at work using conventional 

antenna arrangements and a remote positioning system such as GPS, 

FIGS. 2a and 2b are schematic views of conventional roving and base station 
equipment which may be used by operators such as those in FIG. 1, 

FIGS. 3a and 3b are perspective views of a residential development site and an 
|=Q earth moving operation to demonstrate several environments in which operators may 
work, 

u FIG. 4a is a schematic representation showing general flow of information 

between hardware, software and database components in a preferred embodiment of 

O 

ijl roving apparatus according to the invention, 

15 FIG. 4b is a schematic representation showing general lines of communication 

between hardware components of the preferred embodiment, 
jy FIG. 4c shows a controller device which is part of the apparatus in FIG. 4b and 

may be used by an operator when interacting with the apparatus, 

FIGS. 5a and 5b show alternative head position systems which may be used in 
20 the roving apparatus of FIG. 4b, 

FIGS. 6a and 6b indicate respective geomehrical arrangements of the antennae 
and operator head locations for FIGS. 5a and 5b, 

FIG. 7 is a flowchart indicating generally how a system such as shown in FIG. 4a 
creates an augmented field of view for the operator, 
25 FIG. 8 shows geometrically how a virtual object may be aligned with a real world 

object to create the augmented field of view, 

FIG. 9 is a flowchart indicating how images are calculated for each eye of an 
operator such as shown in FIG. 8 to create a stereo display. 
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FIG. 10 shows a surveyor at work using apparatus according to the invention 
and indicates a visual observation which he or she might make of a site, 

FIG. 11 shows a field of view such as indicated in FIG. 10 including navigation 
symbols as may be displayed for the operator, 

FIG. 12 is a flowchart indicating how the apparatus of FIG. 4a generates a display 
of navigation information for the operator, 

FIG. 13 shows a field of view containing new features and attribute information 
which the operator has input using the real controller device, 

FIG. 14 is a flowchart indicating how attribute information such as shown in FIG. 
13 may be modified, 

FIG. 15 shows a survey operator at work using roving apparatus according to the 
invention to measure the position of a ground point using a virtual range pole, 

FIG. 16 shows an augmented field of view containing a virtual range pole being 
used to collect position data at inaccessible points, 

FIG. 17 is a flowchart indicating how a the preferred roving apparatus obtains 
position data using a virtual range pole such as shown in FIG. 16, 

FIG. 18 shows apparatus including alternative roving apparatus in which a 
virtual interface may be provided for the operator, 

FIG. 19 shows an augmented field of view containing a virtual interface and 
alternative pointing devices, 

FIG. 20 is a flowchart indicating how the apparatus of FIG. 18 may receive input 
from an operator using a virtual interface, 

FIG. 21 shows an operator at work on a building site checking the design of a 
half finished structure, 

FIG. 22 shows an augmented field of view in which an intersection function has 
been employed to calculate and display a result point, 

FIGS. 23a and 23b are augmented fields of view demonstrating entry of detail 
using a virtual interface. 



FIG. 24 is a flowchart indicating how a hinction such as that shown in FIG. 22 
may be implemented, 

FIG. 25 shows an augmented field of view in which an elevation mask and a 
number of satellite positions have been displayed, 

FIG. 26 is a flowchart indicating how an elevation mask function may be 
implemented, 

FIG. 27 is a schematic side view of an operator at work in a machine using 
another embodiment of the apparatus for machine control, 

FIG. 28 shows an augmented view as seen by a machine operator on a road 
construction site, 

FIG. 29 shows an augmented view as seen by machine operator on a pile driving 

site, 

FIG. 30 is a flowchart indicating generally how the apparatus shown in FIG. 27 
creates an augmented field of view for the machine operator, 

FIG. 31 is a block diagram showing an augmented vision system which uses 
wireless commuaications to receive real-time data from a remote source, and 

FIG. 32 illustrates an alternative embodiment of the system shown in Figure 31, 
which has a separate input device. 
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DETAILED DESCRIPTION 

An augmented vision system to generate stereoscopic images for surveying and 
other applications. Note that in this description, references to "one embodiment" or 
"an embodiment" mean that the feature being referred to is included in at least one 
5 embodiment of the present invention. Further, separate references to "one 

embodiment" in this description do not necessarily refer to the same embodiment; 
however, neither are such embodiments mutually exclusive, unless so stated and except 
as will be readily apparent to those skilled in the art. For example, a feature, structure, 
act, etc. described in one embodiment may also be included in other embodiments. 
1§ Thus, the present invention can include any variety of combinations and/ or 

integrations of the embodiments described herein. 
O The present invention is useful in a wide range of survey techniques and in a 

wide range of environments where survey related work is carried out. In this 
;S specification, "surveying" generally includes, without limitation, topographic, 
1^ hydrographic, geodetic, detail, stakeout, site checking and monitoring, engineering, 
'J mapping, boundary and local control work, and machine control. Thus, the term 
jy "survey-related data" is given broad mearung in this specification in accordance with at 
|t least the foregoing examples. Particular environments in which the present invention 
may be useful include land subdivision and estate development, cadastral surveying, 
20 forestry, farming, mining and earthworks, highway design work, road reconstruction, 
building construction, and marine development projects, and all under a wide range of 
weather and ground conditions. Several techniques and environments are described 
herein by way of example only. Further, note that an "operator" or "user", as the term 
is used herein, is not necessarily a surveyor but may be a less extensively trained 
25 individual. 

It will also be appreciated that augmented vision apparatus according to the 
present invention is potentially useful with any remote positioning system which is 
suitable for survey related work, whether satellite or land based. SateUite based 
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systems currently available include GPS and GLONASS. Several sinrularly accurate 
land based radio navigation systems are under development and might also be used, 
such as those which emulate a configuration of satellites over a relatively small 
geographical area for specific purposes. A detailed discussion of surveying techruques 
5 and remote positioning systems is beyond the scope of this specification, which refers 
primarily to GPS based kinematic survey procedures, but without limitation. 

It will also be appreciated that the invention may be implemented in conjunction 
with a wide variety of survey related equipment which is available from a number of 
manufacturers. The size, configuration, and processing capability of such equipment 
1=0 are continually being improved and redesigned. This specification primarily describes 
'2 survey related equipment which is currentiy available from Trimble Navigation 
P Limited in Surmyvale, Calif orrua and augmented vision equipment which is available 
'4 from i-O Display Systems, LLC of Sacramento, California, but yet agam without 
i~ lirmtation. Other equipment commonly used in virtual reality or augmented reality 
IS systems is also described. 

' J For example, this specification primarily describes conventional equipment in 

ju which the antenna, receiver and handheld data collector of a GPS total station are 

H provided as separate items cormected together by suitable cables. A typical stand alone 

receiver and data collector are the Trimble 5700 and TSCl Survey ContiroUer 
20 respectively, coupled to a dual frequency antenna. Another typical data collector is the 
TFCl pen computer which is commonly used for mapping purposes. A data collector 
in this form provides a convenient portable interface by which an operator controls the 
receiver, stores position data and may be guided through parts of a survey related 
procedure. However, receiver devices take many forms and may be incorporated 
25 within the antenna housing, as in the Trimble 4600 for example, or within the data 
collector, by way of a PCMCL\ (Personal Computer Memory Card International 
Association) card in a laptop computer, as in the Trimble PC Card 115. These and other 
arrangements of equipment are also within the scope of the present invention. 
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FIG. 1 shows two survey operators 100 and 110 at work recording position data 
using respective roving apparatus, and receiving remote positioning signals from four 
GPS satellites 120. Operator 100 is using a satellite antenna, receiver and telemetry 
system carried in a backpack 101, controlled by a handheld computer device 102 for 
5 data collection, connected through cable 103. The satellite antenna 104 is mounted on a 
short pole 105, and a telemetry antenna 106 is the only other visible component of the 
system in this view. Operator 110 is carrying a receiver and telemetry device in 
backpack 111, controlled by a special purpose handheld computer 112 through cable 
113. A satellite antenna 114 is mounted on range pole 115 and connected to the receiver 
)§ through cable 116. When not in use, the computer 112 may be clipped to the pole 115 or 
*S the backpack 111. Only a telemetry antenna 117 is visible in the backpack. Operator 
Q 100 is recording position information without attempting to locate the antenna over a 

specific grovmd point, perhaps for municipal mapping purposes. Operator 110 is 
\i recording relatively more accurate information, placing the range pole vertically over a 
)3 groimd point of particular interest, perhaps at a building construction site. The position 
'"-4 of the ground point is then determined from the position of the anterma phase center by 
JJj subtiacting the length of the pole. Their typical measurement accuracy ranges are 1-10 
H m and 1-100 cm respectively, although accuracy varies widely depending on a large 

number of practical factors. They may be recording data in real time or for post 
20 processing, and may be using kinematic or differential teclmiques. 

FIGS. 2a and 2b show typical equipment which might be used in the field by one 
of the operators in FIG. 1, bearing in mind the many alternative arrangements such as 
those mentioned above. FIG. 2a shows roving equipment including a satellite receiver 
200, satellite antenna 201 on pole 202, telemeti-y receiver 203 and antenna 204, and a 
25 data collector and conhroUer 205. The satellite receiver 200 is powered by a battery 
source 206 which may also power the telemetry receiver and the controller if these 
components have no separate power supply. Both the satellite antenna and the 
telemetry antenna /receiver pass data to the satellite receiver for processing along cables 
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as shown, and the results are generally stored in the controller, although they may 
alternatively be stored in the satellite receiver for example. FIG. 2b shows a reference 
base station which is temporarily positioned over a point having a known or assumed 
position, to generate correction data as generally required for measurements made 
5 using kinematic or differential techniques. Fixed reference stations are sometimes 

maintained separately for particular areas by service organizations and need not always 
be set up by an operator. The base equipment includes a satellite receiver 210, satellite 
antenna 211 on tripod 212, telemetry receiver 213 and anterma 215 on tripod 214, and a 
battery pack 216 for the satellite receiver and other components as required. The 
|§ satellite anterma passes data to the satellite receiver for processing, which in turn stores 
'I or passes correction data to the telemetry receiver for transmission to the roving 
O equipment. 

-is, 

'4 FIGS. 3a and 3b show a number of survey and machine operators at work in 

[2 various idealized environments, as separate examples. An augmented vision system 
]3 according to the present invention as will be described below, might be used by each 
y operator in navigating, acquiring data, calculating results, checking work, and so on, 
ju according to the particular need. The examples are intended to convey at least part of 
jt the broad range of work carried out by surveyors and machine operators and are not 
limiting in this regard. They are simplistic but will nevertheless be informative to the 
20 skilled reader. 

In FIG. 3a several residential property areas have been surveyed for 
development at a junction between two streets 300 and 305. A water main 310 has been 
installed for access by houses which may eventually be built on the properties. Each 
street and property area has corner points, boundary lines and other features whose 
25 positions and attributes have already been determined in earUer work and stored as 
database information which is available to the operators. Many of these points will be 
marked by monument pegs. Some of the points are indicated in the figure as small 
circles. The positions of other points and features have yet to be measured, and in 
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many cases the points thennselves will not be ascertained until further development 
takes place. Properties A, B, C, D slope down towards street 300 as indicated by 
contour lines. Properties A and B are rectangles separated by narrow footpaths from 
streets 300 and 305, and each has a supply pipe already laid from the main 310. 
5 Property C has a number of trees, the positions of which are not yet known. Property D 
has a driveway D' to street 300. Both will require a supply pipe from the main 310 on 
either street at some stage. Properties E and F include swampy ground 315 which will 
require some infill and landscaping before building takes place. A broad curved verge 
separates these properties from streets 300 and 305. 

1§ A reference base station 320 such as that shown in FIG. 2b has been set up on 

street 305, to transmit correction data for roving equipment such as that shown in FIG. * 

u 2a, carried by the survey operators in their example tasks. An operator 321 such as 

surveyor 110 in FIG. 1 is navigating along a line joirung points 340 and 341 to record the 

jS elevation of points on the boundary between properties C and D. He may be using 

^15 kinematic, differential or other techniques, and may be recording points as actual 
positions in real time or as raw data for post processing later. Another operator 322 

jy such as operator 100 in FIG. 1 is driving an off-road vehicle over the various properties 
recording data for a map, although in this case the roving equipment may be mounted 
on the vehicle itself rather than carried in a backpack. Operator 323 is searching for the 

20 monument at point 342 which has been overgrown by vegetation, having navigated on 
the site using information presented by the roving apparatus. Operator 324 is recording 
the depth of swampy area 315 at predetermined points to provide an indication of how 
much infill will be required. An approximate volume of infill can be calculated once the 
perimeter and bottom contours of the swamp have been determined. Operator 325 is 

25 staking out an arc between points 343 and 344 to define a curved comer line on one side 
of streets 300 and 305. This is one example of survey calculations which may be carried 
out in the field involving lines, arcs, intersections and other mathematical constructs. 
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In FIG. 3b survey operators carrying roving equipment go about various 
idealized tasks relating to earthnaoving, including road-building, ditch-digging and 
open cast milling, again all by way of example. A number of earthmoving machines are 
also shown with their activity controlled by respective machine operators who work to 
5 guidelines set out by the survey operators. A reference station is typically set up to 
provide correction data for the roving equipment at each site and for the purposes of 
these examples is located in a workers shelter 350. Only the satellite antenna 351 and 
telemetry antenna 352 of the reference station can be seen. A survey operator 360 is 
slope staking the sides of an elevated roadway 380 using measured positions such as 
1=0 381 to calculate desired positions such as 382 to which road fill 383 must be piled. A 
'0 truck 361 supplies road fill material and a bulldozer 362 shapes the material according 
O to directions given to their respective machine operators by the operator 360 or a 

supervisor on the site. Another survey operator 365 is checking the work of an 
!^ excavator 366 in digging a ditch 385. The ditch must be dug by the machine operator to 
1J a required width and depth along a line between points 386 and 387. Finally, a survey 
H operator 370 is determining a cut pattern for an excavator 371 in the bottom of an open 
i|j cast mine 390. A pattern of measured ground points such as 391 is required to ensure 
efficient removal of ore from the mine while maintaining stability of the mine walls 392 
and a spiral road 393. 

20 FIGS. 4a and 4b show the elements of one embodiment of the roving survey 

apparatus which may be carried by a survey operator at work in the field, to provide an 
augmented vision capability according to the invention. FIG. 4a is a schematic diagram 
showing generalized hardware, software and database components of the apparatus 
and connections between them. A rendering system 400 determines the operator's 

25 current field of view by estimating operator eye positions using information from a real 
time head position system 405, a head orientation system 410, and information relating 
to dimensions of the operator's head and the headset. The field of view generally 
contains real "objects" which are being observed in the envirorunent by the operator, or 
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may be hidden from sight, and is augmented with images of virtual "objects" which are 
generated by the rendering system and presented on a display 415. These virtual 
objects include representations of selected physical items and mathematical constructs, 
with associated attribute information. They are typically superimposed by the display 
5 on corresponding real objects in the field of view, such as the physical itenis themselves 
or one or more survey points. The operator controls the apparatus through an interface 
417 which may be partly implemented through the display 415. Position and attribute 
information relating to selected real objects in a particular environment is stored in a 
database 420 which is accessed by the rendering system to generate the corresponding 
|l=0 virtual objects. The database information is generally prepared beforehand from survey 
'I results recorded in the environment during earlier work, or added by the operator 
13 during the current work using an optional but generally desirable data acquisition 
^3 system 425. Other database facilities would also normally be carried by the roving 
\~ apparatus such as an almanac of satellite information. Some example fields of view are 
15 given below. 

H FIG. 4b is another schematic diagram showing an arrangement of currently 

jy available hardware components for the roving survey apparatus. This is one 

embodiment of the invention which incorporates apparatus as previously described 
and shown in FIG. 2a. The rendering system 400 and object database 420 shown in FIG. 

20 4a are provided generally as a separate processor and memory unit 450. The head 
position system 405 is provided by a satellite anterma 455, satellite receiver 456, and 
telemetry antenna /receiver 457, with the satellite receiver connected to the display 
processor 450 by an appropriate cable to pass position data. Head orientation system 
410 is provided by a head mounted sensor 460 again connected to the display processor 

25 by an appropriate cable to pass orientation data. Augmented display 415 is provided 
by a headset 465 and typically receives a VGA signal from the rendering system. 
Boundaries are generally imposed above and to either side of the operator's peripheral 
vision by mechanical components of the headset, and these generally determine the 
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angular extent of the field of view. The operator interface 417 is provided by a 
controller 480 similar to that shown in FIG. 2a and explained further in relation to FIG. 
4c, bearing in mind alternative arrangements as mentioned below. The optional data 
acquisition system 425 is provided by a second satellite antenna 475 and receiver 476, 
5 the telemetry antenna /receiver 457, and a controller 480. New position data obtained 
using the acquisition system is typically processed in the controller before being passed 
to the display processor and memory to be stored in the object database. Attribute 
information relating to the new data or to existing data is entered by the operator 
through the controller for storage in the database. New virtual objects, such as the 

1=0 results of survey calculations that may be carried out by the operator using the 

''^ controller, are also stored in the database as required. 

O The apparatus of FIG. 4b can be provided in a variety of different forms typical 

'J for GPS and other remote positiorung equipment as mentioned above. For example, the 
I R two satellite receivers 456 and 476 which are shown separately may be combined in a 
15 single unit or may be built into the housings of their respective antennas 455 and 475. 
' J The display processor and memory 450 may be combined with the headset 465 or the 
lU controller 480, each of which generally requires a respective processor and memory. In 
one embodiment the display processor and memory, and the controller, can be 
provided together by a handheld or similarly portable computer using a single general 
20 purpose processor and memory for both functions. The receivers 456 and 476 could 
also be included in a portable arrangement of this kind. In some currently available 
equipment the antenna, receiver and controller are combined in a single handheld vinit, 
which is useful for recreational purposes such as hiking or boating. In other 
arrangements, described below, the data acquisition antenna 475 or the controller 480, 
25 or both, are provided as virtual objects, which may be manipulated by the operator as 
result of possibilities created by the present invention. 

FIG. 4c illustrates a handheld controller 480 such as shown schematically in FIG. 
4b, generally similar in appearance to existing devices such as the TSCl. This provides 
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one interface by which an operator may interact with the preferred roving apparatus 
during a survey procedure. An alternative virtual controller system is described below 
in relation to FIG. 17. A partial or fully voice-operated controller system might also be 
used. The controller 480 is an electronic device having internal components such as a 
5 processor, memory and clock which will not be described. Externally the device has a 
multiple line screen 481 such as an LCD, a keypad 482 such as an array of touch 
sensitive buttons, and a number of input/output ports 483 for connection to other 
devices in the roving apparatus. The screen 481 shows by way of a simplistic example, 
a number of high level functions through which the operator is scrolling for selection. 
1^ These include input of operator head characteristics as described below in relation to 
^5 FIGS. 6a and 6b, a navigation function as described in relation to FIG. 11, data 
O acquisition perhaps using a virtual pole collector as in FIG. 15, input of new attributes 
•G for features already existing in the database 420 or recently acquired, alteration of 
in stored data or attributes using a virtual system such as shown in FIG. 14, and a 
15 calibration function by which the operator may adjust an offset in the display 415 to 
'4 align virtual objects more closely with their corresponding real objects if required. 
fU Other functions described below include calculation of intersections and display of 
satellite locations and an elevation mask. Antenna height may also be input by the 
operator. The keypad 482 in this example includes a full set of alphanumeric characters, 
20 function keys, mathematical operation keys, and arrow keys which may be used by the 
operator to indicate calibration adjustments, or alteration of virtual objects and 
information in the display 415. The ports 483 allow input of position data from the 
satellite receiver 476, input or output of database information to an office computer for 
those controllers which contain the display processor and database 450, and other 
25 connections which may be required in practice. 

FIGS. 5a and 5b show alternative headset systems which may be worn by a 
survey operator 500 to provide augmented vision capability according to two 
embodiments of the invention. In each case the headset is based on general purpose 
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head-mounted display (HMD) equipment, such as that available from I-O Display- 
Systems, LLC and described in WO 95/21395 for example. A variety of different 
headsets could of course be used, or manufactured for this particular purpose, and 
much research has been carried out on HMD devices to date. A main component 510 of 
5 the headset contains electronics and optics required to produce a see-through image for 
each eye of the operator, given an appropriate input signal on cable 511. The function 
of this component and the nature of the input signals will be well known or readily 
determined by a skilled reader, such as through the specification mentioned above and 
references therein, so need not be described in detail. Optical combiners 512 and 513 
include a transparent window having generally opaque support components which 
^0 determine the field of view, although the operator may generally look downwards to 
1 3 avoid the window, and obtain a clear view of the controller, for example. The window 

,e 

allows visible light from the environment to reach the eyes of the operator and provide 
li natural images, while simultaneously presenting a generated image for each eye from 
1_5 the main component 510. Light reflected and received from real objects under 

observation by the operator is thereby combined with light generated by the main 
component to create virtual objects and related information superimposed on the 
1^' operators' field of view. Optical combiners 512 and 513 can also be turned off to 
provide the operator with a clear field of view. The virtual objects are generally 
20 displayed in stereo by creating an image for each eye containing similar detail but from 
the slightly different perspective which results from separation of the eyes on the 
human head. This process will described further in relation to FIG. 8 below. 

Other standard components of these headsets include a semi rigid frame 515, 
straps 516 which are adjustable to fit the head of a wearer comfortably and securely, 
25 earphones 517 which may provide sound to accompany the visual images presented on 
the combiners 512 and 513, a head orientation sensor 460, and a microphone if voice 
input is required. Various orientation sensors are available to assist with a head 
tracking function, including inertial, electromagnetic. Hall effect and flux gate devices. 
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as mentioned in WO 95/21395. Their location on the operator's head is not critical, as 
long as the sensor is firmly fastened to the head, and they are shown with two different 
positions in FIGS. 5a and 5b. Each device provides an output signal on cable 521, 
containing yaw, pitch and roll information with reference to a coordinate system 
5 centered within. Devices which can produce angular measurements with an accuracy 
better than 0.1° as generally required in practice are commercially available. The 
function of a suitable head orientation component and the nature of the output signal 
will be well known or readily ascertained by a skilled reader from reference material 
provided with commercially available devices. 

In the embodiment of FIG. 5a, a satellite antenna 550 has been incorporated on 
*| the headset to determine operator head position using signals from a remote 
□ positioning system such as GPS. The antenna is an example of the antenna 455 in FIG. 
-J 4b which passes satellite signals along cable 551 to a receiver device which has not been 
It shown. The head orientation sensor 460 is attached to frame 515 near the operator's 
1^ right temple. In the embodiment of FIG. 5b a satellite antenna 560 is located at a 
'4 distance from the operator's head, typically mounted on a pole 561 carried in a 
iy backpack such as shown in FIG. 1. This antenna generally requires a respective 
12 orientation sensor 565. Satellite signals from the anterma are passed along cable 562 and 
those from the additional sensor 565 along cable 566. The head orientation sensor 460 is 
20 attached to the main component 510 of the headset near the operator's forehead. In each 
figure there is a known geometrical relationship between the satellite antenna 550 or 
560 and the operator's head as will be explained in relation to FIGS. 6a and 6b below. 
Head position and orientation information allow the position of each of the operator's 
eyes to be determined and thus the operator's field of view. An alternative 
25 arrangement involves three or more small satellite antennae attached to the headset to 
provide both head position and orientation data from the remote positioning system 
without need of the separate orientation sensor 460. 
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FIGS. 6a and 6b indicate simple mathematical models for calculating operator 
eye positions given head position and orientation information from the headset systems 
shown in FIGS. 5a and 5b respectively. This allows the rendering system 450 in FIG. 4a 
to determine a direction for the operator's instantaneous field of view F and therefore 
which virtual objects can be presented on the display 415. Some geometric information 
giving the position of each eye with respect to the antenna 550 or 560 is also required, 
stated as distances in three dimensions between the phase center of the particular 
antenna and the center of the operator's eyeballs. Forward, transverse and vertical 
distances with respect to the operator's head are designated as parameters x, y, z 
respectively and are added or subtracted from the antenna position by the rendering 
system as required. For accurate survey work the geometric information may be 
determined and input to the roving apparatus using individual characteristics of the 
particular operator, and in circumstances with less demanding requirements such as 
mapping or design checking, may be approximated by standard characteristics of a 
male or female head and neck. A dynamic calibration option will also normally be 
provided in which a selected virtual object in the display is aligned with a 
corresponding real object visible to the operator when the headset is initially placed on 
the head. Occasional calibration checks will also normally be performed by an operator 
at work to detect whether the headset has moved from the initial placement. 

In the embodiment of FIGS. 5a and 6a the antenna 550 is located directly on top 
of the operator's head 600 once the headset is put in place, and moves with the head as 
the operator looks in different directions. For an upright head the operator's field of 
view F may be taken as originating at a pair of eyeballs 601 positioned a distance xl in 
front of, and zl below the antenna position, separated sideways by a distance yl. These 
distances are assumed to be constant in the absence of any relative movement between 
the headset and head. Typical values for these parameters on a human head are xl=10 
cm, yl=6 cm, zl=12 cm. For a head oriented away from upright by yaw, pitch and roll 
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angles -y -p -r the actual distances between antenna and eyeballs are readily calculated 
by matrix multiplication as follows: 



1 0 0 

0 cos^9^ -sin^^ 
0 sinfl?^ cos(p^ 



sin^p cos(Pp 0 
0 0 1 



COS(py 

0 

^sin(Py 



0 -sin^JyVx^ 



1 
0 



0 



5 In the embodiment of FIGS. 5b and 6b, the antenna 560 is located behind the 

operator's head 600, mounted on pole 561, and does not generally move as the head 
turns to look in different directions. Calculating the operator eye positions from the 
anterma position in this case is a two step process of determining distances x2, y2, z2 
Iq from the antenna to a fixed point 602 at the top of the neck, about which the head is 
assumed to pivot, and distances x3, yl, z3 from point 602 to the eyeballs 601. Typical 
values for these parameters in relation to a himian head are x2=20 cm, y2=0, z2=30 cm, 
x3=16 cm, z3=18 cm. However, the antenna will not necessarily remain upright, as the 
operator bends forward for example, or imdergo the same changes of orientation as the 
operator's head. Both the head and antenna therefore require respective orientation 
sensors 460 and 565. The system of FIGS. 5b and 6b is more complex and prone to error 
than that of FIGS. 5a and 6a, as for example, the backpack which holds the antenna 
must be attached firmly to the operator so that distances x2, y2, z2 remain suitably 
constant. Whether or not a less preferred system in this form is used in practice will 
depend on whether the accuracy of aligrunent between real and virtual objects in the 
20 augmented display is acceptable imder the circumstances. 

FIG. 7 is a flowchart which broadly outlines a routine which is continuously 
repeated by software in the rendering system 400 of FIG. 4a to create an augmented 
display 415 for the operator in real time. In step 700 the renderer first gets a current 
position measurement from the head position system 405, such as a measurement of 
25 antenna 455 generated by receiver 456 in FIG. 4b. The renderer may also require an 
orientation measurement for the antenna in step 705, such as a measurement from 
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sensor 565 when the operator is using a system as shown in FIG. 5a. A measurement of 
operator head orientation is required from system 410 in step 710, such as output from 
sensor 460. In step 715 the renderer can then calculate operator eye positions and a field 
of view according to a geometrical arrangement of the antenna and head as shown in 
FIG. 6a or 6b. Information relating to the position, shape and attributes of virtual 
objects which are to be displayed is then obtained from database 420 in step 720. 
Finally an image is generated for each eye using the database information, and optional 
input from the operator as explained below, and passed to the headset 465 for display 
in step 725. More detail on this last step is given in relation to FIG. 9 below. 

The latency or speed with which the display may be updated in this routine as 
the operator moves and looks about an environment is limited primarily by the speed 
and accuracy of head position measurement. Real time measurements accurate to about 
1 cm or less can be obtained by available receiver equipment at a rate of about Is each. 
Measurements accurate to orJy about 2 cm generally require less time and can currently 
be obtained in about 0.2s each. The operator may be required to be make more or less 
deliberate movements depending on the accuracy which is acceptable in particular 
circumstances. Predictive techniques may be used to reduce latency if required but are 
beyond the scope of this specification. Some discussion of systems for predicting head 
positions in advance is found in the article by Azuma and Bishop mentioned above. 
The degree of misregistration between virtual and real world objects depends on 
various factors, including the accuracy of contributing position and orientation 
measurements in FIG. 7 as mentioned above, and on the distance at which the virtual 
object must appear to lie. There are also usually errors in the headset optical systems. 
Misregistration is more or less tolerable depending on the operator's requirements. 

FIG. 8 is a diagram to illustrate simply how a virtual object is generated in stereo 
by the rendering system 400 in FIG. 4a, to correspond with a real object in the operator's 
field of view. In this example the operator's left and right eyes 800 and 801 are looking 
through semi-transparent display devices, such as optical combiners 512 and 513 of a 
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headset 465, towards a tree 805 at a distance Dl. Information relating to the tree is 
stored in database 420, such as the actual position of two points 806 and 807 on trunk 
810, connected by a dashed line, and a point 808 at the top of the tree. An attribute such 
as the type of tree may also be included. The renderer calculates left and right eye 
5 images on a plane area 820 at a prescribed distance D2, along respective lines of sight to 
the tree, as will be described in relation to FIG. 9 below. A calculation in this form is 
typically required by available headsets for processing and output of images on the 
combiners to create a stereo display. The images are shown generated as dashed lines 
825 and 826, each aligned with trunk 810, to create a corresponding virtual object for the 
1£l operator as a single dashed line 827 fully within the field of view. Simple images of this 
^2 type are generally sufficient for most purposes, and other parts of a real object such as 
O branches of the tree 805 may or may not be represented in the corresponding virtual 
^5 object. Other significant points on the real object such as tree top 808 will in some cases 
™ be recorded in the database but lie outside the field of view, generally on a line which 
15 lies outside the plane area 820, and not capable of representation. 

FIG. 9 is a flowchart which broadly outlines a routine which may be 
i\ i implemented during step 725 of the routine in FIG. 7, to generate two images such as 
j J shown in FIG. 8. In step 900 the rendering system 400 determines the plane containing 
area 820 at a perpendicular distance D2 in front of the operator's eyes 800 and 801. The 
20 plane is characterized by an equation in a local coordinate system, generally the system 
to which the object position data is referred. This involves standard, known 
mathemahcal operations which need not be described herein. In step 905 lines are 
determined joirung the center of each eye to each point on the real object which is 
recorded in the database 420, being lines to points 807, 806 and 808 at distance Dl in this 
25 example. The intersections of these lines with the plane are then calculated in step 910, 
indicated by crosses. Given the intersection points, step 915 then determines image 
points and lines, and other features for display, having characteristics which may be 
specified in the database, such as dashed lines 825 and 826. Any lines or points which 
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lie outside area 820 are clipped in step 920, and any attribute information from the 
database is presented to fit on area 820 in step 925. Finally details of the image are 
passed to the headset 465 for display, and any further processing which may be 
required. 

5 FIG. 10 shows a scene in which a survey operator 140 wearing roving apparatus 

according to the invention has a field of view 145 containing several real objects which 
have virtual counterparts. The field of view 145 is indicated as an approximately 
rectangular area roughly equivalent to area 820 in FIG. 8. This operator is wearing a 
headset 465 such as shown in FIG. 5a and carrying a satellite antenna 475 on a range 
pQ pole for data acquisition which may be required on this site. A controller 480 is clipped 
'5 to the pole. A small tree 150, survey monument 151, one edge of a concrete path 152, 
^3 and part of an undergroimd main 153 including a branch 154 are within the field of 
'4 view. A corresponding virtual object is presented to the operator using stored image 
jS features and attributes, somewhat unrealistically in this figure for purposes of 
JL^ explanation, as only a single target object of interest to the work at hand would 
'^-1 normally be presented at any one time. Another monument 155 and another branch 

1=5: 

lu 156 from the main are outside the field of view. The operator in this example could be 
doing any one of several things, such as checking whether tree 150 still exists, locating 
and checking the position of monument 151 which may not have been surveyed for 

20 many years, staking out additional points to determine the edge of path 152 more 

precisely, or placing a marker for a digging operation to repair branch 154 in the water 
main. In each case he must navigate to a target point on the site to take a position 
measurement or carry out some other activity. 

FIG. 11 shows the augmented field of view 145 as might be observed by the 

25 operator 140 in FIG. 10, once again including more target objects than would normally 
occur in practice. In this example the position of monument 151, which is recorded in 
the object database with a code "M99", is shown marked by a virtual flag, although the 
monument itself is missing and will need to be replaced by the operator. The 
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underground main 153 carmot be seen although target branch 154 coded "B12" can be 
located and marked. Navigation symbols 160 and 161 are presented in the display to 
indicate the positions of monument 155 and branch 156 recorded as "MlOO" and "Bll" 
respectively. They indicate to the operator a direction in which to look or walk in order 
5 to locate the real object targets, without needing to deterniine a compass direction, as 
will be evident. The symbols may take various colors or flash if required. It is assumed 
here that the operator has an interest in each of the real objects which have been shown, 
and has caused the display of a corresponding virtual object or navigation symbol in 
each case. In general however, the display would be considerably simpler if the 

iKI operator was concerned with a single object. The work of operator 140 in FIGS. 10 and 
11 may be regarded as generally comparable to the operators in FIG. 3a such as 

□ operator 323. 

^5 FIG. 12 is a flowchart which indicates how the rendering system 400 in FIG. 4a 

generates navigation symbols in the display on request by the operator, such as those 
shown in FIG. 11. The operator first indicates a target point of interest, typically 

'si through the controller 480 in FIG. 4b by entering a code such as "MlOO". In step 230 the 

U 

iil rendering system 400 receives this code from the controller, and obtains information 

2 regarding the target point from the object database 420 in step 235. The renderer must 
then determine the current field of view as in FIG. 7, and in step 240 obtains the 

20 operator head position and orientation from systems 405 and 410 to carry out the 
calculation. If the target point is already within the field of view a virtual object is 
created and displayed in step 245. Otherwise in step 250 the renderer deternunes 
whether the target is up, down, right or left from the field of view and creates an 
navigation symbol in the display indicating which direction the operator should turn, 

25 typically in the form of an arrow. The routine continues to determine the current field 
of view and either present a virtual object corresponding to the target in step 245 or 
update the navigation symbol until halted by the operator. Other navigation 
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information may also be presented such as distance and bearing to the particular real 
object to which the operator is seeking to move. 

FIG. 13 shows another augmented field of view of somewhat idealized work in 
progress, which might be seen by an operator using roving apparatus according to the 
5 invention. This example demonstrates input of information by the operator using a 
virtual cursor 650 which could take many shapes. The operator is observing a ditch 680 
dug by an excavator to reveal an electricity cable 681 and a water main 682, in a similar 
context to operator 365 in FIG. 3b. Points at various positions along the cable and water 
pipe have been surveyed in earlier work and are already in he database with code and 
1=Q attribute information. Virtual objects corresponding to these real and visible objects are 

'2 indicated as dashed lines 655 and 656 respectively, with an appropriate attribute 

iy 

13 "ELECTRICITY" or "WATER". Points 670, 671 on the cable and within the field of view 

are indicated by virtual markers coded "El", "E2" and could represent power feeds 
H which have not been shown. Points 672, 673 on the water main are similarly indicated 

1 5 by virtual markers coded "Wl", "W2". A gas main is to be laid parallel to the existing 

(□ 

'==4 features and the operator has determined the position of two further points 674, 675 at 
\% I which a gas pipe will be placed. Virtual prompt markers are shown at these points and 
!^ the operator may now use the controller 480 in FIG. 4b to move the cursor 650 to 

separately select the markers for input of respective codes, such as "Gl" and "G2". The 
20 operator has already created a dashed line 657 between points 674, 675 as a virtual 

object representing the gas pipe. An attribute for the object may now be input as also 

prompted, predictably "GAS". 

FIG. 14 is a flowchart indicating for input of database information using a virtual 

cursor such as shown in FIG. 13. The operator first selects an input option on the 
25 controller 480 such as shown on screen 481 in FIG. 4c. The rendering system 400 then 

calculates the field of view in step 750 as previously described. A virtual cursor is 

created in the display at a start position such as the lower right comer of FIG. 13, by 

step 755. Operator input at the controller through the arrow keys on keypad 482, 
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indicates incremental shifts for the cursor in the display in a loop formed by steps 760, 
762 and 764. An equivalent effect could be produced by holding the cursor at a fixed 
location in the display and having the operator make head movements to vary the field 
of view. After moving the cursor on the display the operator may select a desired item 
5 in step 764, such as one of the prompts in FIG. 13, or an existing attribute for alteration. 
An option to create a virtual object such as a dashed line between existing points is also 
provided and may be selected by appropriate positiorung of the cursor and button on 
the controller. An option to delete items is similarly provided. The renderer then waits 
for an input from the controller keypad in step 770, and presents the input in the 
|jO display for viewing in step 775. Once satisfied with the input which has been presented 
or any changes which have been made the operator may store the new information in 
database 420 as required in step 780. The cursor is removed when the routine is halted 
by the operator. 

Id A data acquisition system 425 for the preferred roving apparatus shown in FIGS. 

in 

4 5 4a and 4b can be implemented in several ways depending on the accuracy of position 

in 

measurements which are required. An operator can collect position information at 
points of interest in conventional ways as mentioned in relation to FIG. 1, using either 
the physical range pole 475, anterma 474, and receiver 476, similarly to operator 110, or 
using the head position anterma 455 and receiver 456, similarly to operator 100 and with 
20 generally less accurate results. Either kinematic or differential techniques may be used, 
and because the rendering system 400 requires real time measurements from the head 
position system 405 to generate the augmented display 415, data acquisition also 
produces real time position coordinates rather than raw data for post processing later. 
The present invention enables information to be collected using either of these 
25 arrangements in real time with an optional measurement indicator presented as a 
virtual object in the display 415, as will now be described. 

FIG. 15 shows a scene in which a survey operator 740 is measuring the position 
of point 760 at one comer 761 of a house 762 using one embodiment of the roving 
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apparatus according to the invention. The field of view 745 is indicated as an 
approximately rectangular area roughly equivalent to area 820 in FIG. 8. This operator 
is wearing a headset 465 with antenna 455 such as shown in FIG. 5a, and carrying a 
satellite antenna 475 on a range pole 474 for the data acquisition system 425 in FIG. 4a. 
5 It is not possible to place the range pole exactly at the corner 761 and directly take a 
useful measurement of point 760 for several general reasons which arise from time to 
time in survey activities. In this case the physical size of the antenna prevents the range 
pole from being oriented vertically over the point of interest, and the house structure 
prevents the antenna from receiving a sufficient number of satellite signals. The house 

II) structure may also generate multipath reflection signals from those satellites which do 
remain visible to the antenna. Practical problems involving physical inaccessibility or 

O lack of signal availability such as these are normally solved by measuring the position 
of one or more suitable nearby points and calculating an offset. The operator here 

;3 makes use of a virtual range pole or measurement indicator 750 which may be created 

Ms 

2] 5 anywhere in the field of view by the rendering system 400 in FIG. 4a. This virtual object 
is shown in dashed form as a semi circular element on top of a vertical line which 

jy resemble the antenna 475 and pole 474, although an indicator could be presented in 

p various ways such as a simple arrow or flashing spot. 

The position of virtual pole 750 is determined as an offset from that of antenna 

20 475 or antenna 455 in the system of FIG. 5a or 5b respectively. The position of virtual 
pole 750 and its appearance in the field of view may be adjusted as required by the 
operator. Antenna 475 is generally to be preferred because the operator can more 
readily hold pole 474 steady for a few seconds or more as required to make an accurate 
measurement using currently available receiver equipment. 

25 Antenna 474 moves with the operator, and particularly in the system of FIG. 5a 

moves with the operator's head, so is less likely to remain steady for the required 
interval and will generally produce a less accurate position measurement. The operator 
may look downwards at a controller 480, for example. However, either arrangement 

28 



m m 

may be used in practice depending on the level of accuracy required in the work being 
carried out by the operator. Accuracy also depends on correct calibration in the 
alignment of virtual and real objects, and the distance at which a measurement using 
the virtual pole is sought. Submeter accuracy is generally possible using a virtual pole 
5 offset by up to around 5 m from antenna 475 carried separately on a real range pole. 
Improvement in the speed of available equipment is expected to improve the 
acceptability of measurements made using antenna 455. 

FIG. 16 shows an augmented field of view containing a virtual range pole 750 as 
might be used by an operator to record position information at one or more inaccessible 
points according to the invention. In this example the operator is standing on one side 
^2 of a river 840 measuring the positions of two trees 845 and 846 on the other side, and 
ID also the height of a ledge 851 on a nearby bluff 850. A position has already been 

c 1= 

^Lj measured for tree 845 and stored in the database 420 along with a corresponding virtual 

i ^ 

object which now appears as dashed line 855. The virtual range pole is shown 
1J approximately centered in the field of view and may be moved to tree 846 or the ledge 

851 by the operator using controller 480, or a virtual controller as will be described 
ly below in relation to FIG. 18. Should the operator choose to look elsewhere in the 

1^2 environment during this process the pole may fall outside the field of view and will 

J— 

disappear from the display. On looking back across the river the virtual pole returns at 
20 one or other side of the display. Alternatively, a reset function on the controller could 
be used to replace the pole in a central position in the field of view. 

FIG. 17 is a flowchart indicating a routine by which the rendering system 400 
may enable position measurements to be recorded using a virtual range pole such as 
shown in FIG. 15. The operator first selects data acquisition as an option on the 
25 controller 480 as shown in FIG. 4c. Rendering system 400 then calculates the field of 
view in step 950 as previously described. The current position of anterma 455 or 475 is 
obtained in step 955. A virtual pole is then created at a start position such as the center 
of the display in FIG. 15, by step 960 . Operator input at the controller indicates 
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incremental offsets for the pole, and eventually stores a position measurement in 
database 420 in a loop formed by steps 965 to 985. In step 965 the renderer waits imtil 
the operator indicates an offset, such as through the arrow keys on keypad 482, and 
then calculates the new pole position in step 970. The pole can then be recreated in the 
5 display at the new position in step 975. Each push of an arrow key moves the pole a 
fixed angular distance in the field of view for example, and holding the key down 
causes the pole to move continuously. The operator indicates through the controller in 
step 980 when the position of the virtual pole is to be stored as a point in the database, 
or may otherwise terminate the routine to remove the pole from the display. On 
10 storing a new point the renderer may also create a virtual object in the database such as 
^0 flag 855 in FIG. 15 and present the object in the display as confirmation that the 
□ measurement has taken place. FIG. 18 shows the apparatus of FIG. 4b in which 
Ti controller 480 has been optionally replaced by pointing and sensing devices 490 and 491 

H which may be used with the headset 465 to provide an alternative interface for the 

III 

^1 5 operator. A variety of pointing and sensing systems are known, such as the glove 
'y system described in U.S. Pat. No. 4,988,981 produced by VPL Research Inc., and need 
1 7: not be described in detail herein. Another possible pointing device is a pen or wand as 
P known in virtual reality technology. The operator wears or carries the pointing device 

490 with one hand and the display processor 450 produces a virtual control object in the 
20 field of view which resembles or is equivalent to the controller 480, as described in 

relation to FIG. 19. The pointing device has an indicating component such a finger tip 
on the glove, or the pen tip, which the operator sights through the headset and aligns 
with desired inputs on the virtual control object. The sensing or tracking device 491 
may be located on the headset 465 or elsewhere on the operator such as on a belt. It 
25 continuously determines the position of the indicating component and thereby any 
inputs required by the operator. 

Various methods may be used to sense the position of the pointing device and 
the indicating component in front of the headset. One such method makes use of a 
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Polhemus 3D tracking system such as that available under the product name 3SPACE 
INSIDETRAK. According to this method the tracking device 491 includes a small 
transmitter that emits magnetic fields to provide a reference frame. The pointing device 
includes a small receiver that detects the fields emitted by the transmitter and sends 
information to a processor system for analysis. The processor system calculates the 
position and orientation of the receiver and thereby the pointing device. 

FIG. 19 shows an augmented field of view containing a virtual control object 940 
and alternative pointing devices which might be used with roving apparatus according 
to the invention. In this example the operator is using a virtual range pole 945 as 
described above in relation to FIG. 15 to measure the position of point 961 at the base of 
a tree 960. Control object 940 is created by the rendering system 400 to resemble 
controller 480 in FIG. 4c although many features of the keypad 482 have been omitted 
here for clarity. The pole has been offset to the tree position and the operator may now 
indicate that a position measurement as shown in the screen 481 be stored. One 
alternative pointing device is a glove 970 having a Polhemus receiver 975 located on the 
index finger 973. Another possible pointing device is pen 980 having a Polhemus 
receiver 985. Information from the receiver 975 or 985 is passed from each pointing 
device along respective cables 971 and 981. The tips of the index finger and the pen are 
indicating components which the operator positions at appropriate keys of the virtual 
control object for a predetermined length of time to select a desired input for the 
rendering system. A push button on the pointing device may also indicate when an 
input is to be made. Confirmation that the input has been successfully input may be 
provided as an indication on screen 481 or by highlighting the key on keypad 482 which 
has been selected. 

FIG. 20 is a flowchart outlining broadly a routine by which the rendering system 
400 may provide an interface for the operator through a virtual control object such as 
shown in FIG. 19. The operator first indicates to the renderer in step 990 that the control 
object should be created in the display, through a push button on the pointing device 
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for example. This could also be achieved by simply raising the pointing device 490 into 
the field of view. The control object is then created in step 991 and the position of the 
indicating component of the pointing device is monitored for acceptable input in a loop 
formed by steps 992, 993 and 994. In step 992 the renderer receives the position of the 
5 indicating component from the sensing device 491. This position in relation to the 
headset or to a belt system is converted to a position on area 820 in FIG. 8 and 
compared with those of a set of active regions on the control object, such as the keys in 
step 993. If an active region has been indicated the renderer then highlights the region 
and checks that the indicating component is held in place by the operator for a 
jLp minimum period of time in step 994, typically about one second. Other methods of 
^? checking the operator's intent regarding input at a particular region, such as detecting 
O gestures may also be used. Finally in step 995 the renderer acts on the acceptable input 
'5 and may provide confirmation in the display that a corresponding event has taken 
|s place. 

'IS FIG. 21 shows a scene in which an operator 1100 is working on a site 1110 

\j 

'4 inspecting construction of a building 1120 using roving apparatus according to the 

jy invention. In this example the building is a house and garage, although structures of all 

!3 kinds, including civil, commercial, industrial and other designs as generalized above 

may be visualized. The operator is not necessarily a surveyor but could be a builder or 
20 engineer, for example. Various points on the site have been surveyed in previous work 
and included in an object database which forms part of the roving apparatus. These 
points include monuments 1111, corners of the foundation 1112, a tree 1113 and a 
branch 1114 for an underground utility service such as electricity or water. Parts of the 
building such as some wall and roof structures 1121 and 1122 of the living area are 
25 already partially completed. Construction is yet to begin on other parts such as a 
garage 1123. Virtual objects 1131, 1132, 1133 and 1134 indicating the positions of the 
monuments, foundation corners, the tree and utility branch are also included in the 
database and are presented to the operator as they fall within the field of view. A 
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collection of virtual objects 1135 are included to represent the walls, roof and other 
features of garage 1123. In general, there will be a range of features of the design 
contained in the object database, including points, lines, surfaces and various attributes 
such as those discussed in relation to preceding figures. The operator's inspection of 
5 site 1110 and the building under construction is thereby enhanced by an augmented 
view of some or all parts of the structure. Those parts which are partially completed 
can be checked for accuracy of workmanship. The corners of walls 1121 must align with 
virtual objects 1132 for example. Those parts such which have not yet been started can 
be readily visualized. An outline of the garage 1123 can be seen in a finished form for 
ijQ example. New survey points for additional structures or corrections can be added to 
*S the database during the inspection if required, using methods as described above. 
P FIG. 22 shows an augmented field of view presenting the result of a survey 

calculation which might have been required on site, by operator 140 in FIG. 10 or 
\i operator 1100 in FIG. 21, for example. This optional function of the apparatus produces 
1J the position of an unknown intersection point 1150 determined by two known points 
'4 1151 and 1152, and respective bearings or azimuths 1161 and 1162 from the known 
jy points. All three points are shown in the field of view for purposes of explanation, 
p although in practice they may be further apart so that only one can be viewed at any 
time. Each of the known points 1151 and 1152 are either already stored in the object 
20 database 420, perhaps as the result of earlier calculations, or are measured using data 
acquisition system 425 when required by the operator. The bearings are typically 
entered through interface 417 when required, as will be described below. The 
calculation can be presented to the operator in various ways using virtual objects such 
as those shown. In this case the known points 1151 and 1152 are displayed as flags 1155 
25 and 1156 carrying their database codes "PTIOO" and "PT105" while the unknown point 
1150 is displayed as a flag 1157 coded as "PTX". A numerical code is allocated to the 
unknown point when stored in the database by the operator. Line objects 1163 and 1164 
are optionally displayed according to the required bearings 1161 and 1162 input by the 
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operator. Numerical information stating the coordinates and bearings, for example, 
may also be presented in the field of view, although this may be avoided to ensure 
clarity for the operator. 

FIGS. 23a and 23b indicate how known points 1151 and 1152 and bearings 1161 
5 and 1162 may be selected or input by an operator to form the basis of a calculation such 
as presented in FIG. 22. The example calculation is once again an intersection of two 
lines determined by two points and two bearings, sometimes referred to as "intersection 
of bearings", by way of example. Intersection of two circles or a line and a circle are 
other possibilities, and other functions such as calculation of offsets or inverses would 
%D also normally be provided. Some intersection functions such as that of a line and a 
•'0 circle, produce two possible resulting points. The operator is able to select either in the 
(3 field of view arising a virtual cursor. FIG. 23a shows a data input screen of the operator 
in interface 417 which may be presented on manual controller, such as controller 481 in 

FIG. 4c, or on a virtual controller such as shown in FIG. 19. A virtual data input screen 
'15 is shown in this example. The operator has specified known points coded "PTIOO" and 
'4 "PT105" as inputs "point 1" and "point 2" required by the screen, and has input bearings 
jy "170°" and "70°" respectively to determine the intersection. Selecting "CALC" produces 
!^ a result screen as shown in FIG. 23b. The operator is now presented with northing, 

easting and elevation distances relative to his present position for the intersection point 
20 "PTX". The new point could also be presented as a distance and bearing from the 
present position. Selecting "STORE" stores the point in the database with an 
appropriate code. Selecting "DISPLAY" presents a view such as that shown in FIG. 22. 

FIG. 24 is a flowchart which broadly outlines a routine by which the rendering 
system 400 may provide a calculation function for the operator, such as the intersection 
25 of azimuths function described in relation to FIG. 22. The operator first indicates to the 
Tenderer in step 1170 that a function is required, by selecting an option on the manual 
or virtual controllers shovm in FIG. 4c or FIG. 19, for example. Details are then input by 
the operator in step 1172 using input screens such as those shown in FIGS. 23a and 23b. 
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The tenderer then accesses the object database to check and obtain position information 
relating to the input in step 1174. Information is presented to the operator and the 
required calculation takes place in step 1176. The renderer also calculates the current 
field of view as previously described, and if required by the operator, generates images 
5 for the see through display as shovm in FIG. 22 in a loop formed by steps 1178 and 1180. 
The operator may request storage of the result of the calculation in step 1182 and the 
routine may be ended or the calculation may be repeated with different input. 

FIG. 25 shows an augmented field of view demonstrating a fimction by which 
the location and acceptability of signal sources in a remote positioning system, such as 
^ satellites 120 in FIG. 1, can be indicated to the operator. Satellite signals originating 
^0 below a mirumum elevation are usually ignored by the roving apparatus due to 
P atmospheric effects which degrade signal quality. A mask angle of about 13-15° is used 

i 

^ J by default or may be selected by the operator depending on the number of satellites 
1% available for a position measurement and the precision required in the measurement. 
■1^ In this case the operator is looking towards the horizon 1200 and virtual objects 
' J indicating the minimum elevation and the location of two satellites in the field of view 
jy have been presented in the display 415. A mask angle of 13° is shown in a box 1206 and 
H the minimum elevation is indicated by a dashed line 1207. One of the satellites coded 

"S9" lies in a solid angle indicated by a circle 1211 and is moving relative to the operator 
20 in a direction indicated by arrow 1216. It is currently below the minimum elevation line 
1207 but is moving higher. The other satellite "S13" indicated by a circle 1210 is above 
line 1207 and also moving higher in a direction indicated by arrow 1215. Information 
related to the current elevations and expected positions of these two satellites, or 
summarizing all of the satellites above the horizon, could be presented on the display to 
25 assist the operator. The other satellites would be revealed to the operator by a scan 

around the horizon or upwards towards the zenith. It will be appreciated that the view 
shown here is given from the operator's viewpoint, and that satellite information could 
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be presented by other views such as a vertical section through the operator and zenith, 
or a horizontal section centered on the operator. 

FIG. 26 is a flowchart which broadly outlines how the rendering system 400 may 
indicate the availability of signal sources to an operator using an augmented field of 
5 view such as shown in FIG. 25. In step 1220 the operator first indicates to the roving 
apparatus that a mask related display is required. The required mask angle is then 
retrieved from stored information by the renderer in step 1222, or entered by the 
operator. Access to an almanac of satellite information is then required at step 1224 in 
order to calculate current satellite locations and related data in step 1226. The renderer 
ID next determines the operator's current field of view as already described in detail 
*i above, and generates images which indicate the mask elevation and those satellites 
d which are within the field of view in steps 1228 and 1230. Steps 1224 to 1230 from a loop 
;fl which continually updates the display as the operator's field of view changes. 

^3 FIG. 27 is a schematic diagram showing elements of a further embodiment of 

1 1 1 

d 5 apparatus according to the present invention, providing augmented vision capability 

Q for a machine operator. In this embodiment an operator 1300 is shown working from 

jj^ the cab 1305 or control point of a machine 1310, typically a vehicle such as a truck 361 or 

^3 excavator 366 as shown in FIG. 3b. However, the range of machines and the purpose to 

f _L 
f™ 

which they are put is not limited in this regard. The apparatus contains hardware, 
20 software and database components which are generally similar to those of FIG. 4a 

although some differences result from the operator placement on a machine. A display 
processor and memory 450. containing a rendering system 400 and object database 420, 
and a headset 465 containing an augmented display 415 are provided. An operator 
interface 417 which may be manual or virtual, or enabled in some other form such as 
25 voice control, is also generally provided. However, the real time head position and 
orientation systems 405 and 410 may include a tracking system such as the Polhemus 
3D devices menhoned above, for converuence in determining the position and 
orientation of the operator's head with respect to the machine. In this embodiment a 
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satellite antenna 1320 is carried by the machine mounted on a pole 1321 or directly on 

the machine. This antenna requires an orientation sensor 1325 to account for motion of 

the machine, similar to the motion of the backpack described in relation to FIG. 5b. 

Satellite signals from the antenna are passed along cable 1322 to a satellite receiver 1340 
5 in or on the body 1306 of the machine, for signal processing, and from the receiver to 

the display processor along cable 1341. Signals from the vehicle orientahon sensor 1325 

are passed on cable 1326 to the display processor. 

The position of the head of operator 1300 may be determined in various ways, 

preferably by using a tracker transmitter 1360, tracker receiver 1363 and tracker 
W processor 1366. Transmitter 1360 mounted on the machine emits a magnetic field with 
'0 provides a frame of reference for the receiver 1363 mounted on the operator's head. The 
13 receiver 1363 detects the magnetic fields emitted by the transmitter 1360 and sends 
°h information to the processor 1366 for analysis. The reference frame provided by the 

transmitter 1360 is itself referred to the position determined by the antenna 1360 

111 

15 through a known geometrical relationship of these components on the body of the 

in 

machine. A tracker system of this kind is available under the product name 3SPACE 
i J INSIDETRAK as mentioned above in relation to FIG. 18. Other fields may also be 
p emitted by the transmitter to provide a reference frame such as those in ultrasonic or 

ophcal based systems. Other processor arrangements may also be envisaged in which 
20 the tracker processor 1366 and display processor 450 are combined for example. It will 
be appreciated in general that various alternative systems for determining the position 
and orientation of the machine and the position and orientation of the operator's head 
may be devised. One combined position /orientation system which might be used for 
the machine is the TANS Vector GPS Attitude System, available from Trimble 
25 Navigation Ltd., in which an array of four satellite antennae produce three axis attitude 
and three dimensional position and velocity data. This replaces the single antenna 1320 
and orientation sensor 1325. An alternative position/orientation system for the 
operator's head would be a mechanical head locator, by which the operator must place 
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his or her head in a predetermined fashion in a headrest, for example, with the headrest 
having a known geometrical relationship with respect to the antenna 1320. This would 
replace the transraitter 1360, receiver 1363 and processor 1366 system. 

FIGS. 28 and 29 are augmented fields of view demonstrating envirorunents in 
5 which a machine operator as described in relation to FIG. 27 might be at work. Other 
environments and fields of view are shown in FIGS. 3a, 3b, and FIGS. 11, 13, and it will 
be appreciated that these are all given only as examples. FIG. 28 shows an embankment 
1400 through headset 465, which is to be cut away to form the shoulder of a road 1405. 
The layout of the road has been determined in previous survey and design work, and 
1=0 the required survey points, virtual objects and attribute information have been stored in 
'5 a database of features, as previously described. The machine operator views the 
O embankment through the headset and sees the road design in a virtual form 

superimposed on the existing earth formation. Concealed features to be avoided such 
H as pipes and cables may also be indicated as virtual objects. The work involves 
13 removing earth from the embarJ<ment using an excavator to form a surface indicated by 

i 

t ^ 

a dashed curve 1410, vertical lines 1411 and horizontal lines 1412. A real tree 1415 is 
j.^ flagged for removal with a virtual "X". FIG. 29 shows a set of pile positions as seen by a 
piling machine operator through the headset 465. The piles 1420 are being put in place 
to form the foundation of a building or support for a wharf, according to survey point 
20 positions which have been determined and stored in the object database 420. The 

medium 1430 between the piles is earth or water respectively in these examples. Piles 
1425 have already been put in place and their positions are marked by virtual lines 
1426. Other piles are yet to be placed at positions marked by virtual flags 1427. The 
operator guides the piling machine into position to drive home the remaining piles 
25 where required. 

FIG. 30 is a flowchart which broadly outlines a routine which is continuously 
repeated by software in the rendering system 400 to create an augmented display for 
the operator 1300 in FIG. 27. In step 1450 the renderer first gets a current position 
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measurement for the machine from antenna 1320 and receiver 1340. An orientation 
measurement will also normally be required from ser\sor 1325 in step 1452, in order to 
determine the position of the tracker transmitter 1360 with respect to the antenna 1320. 
Transmitter 1360 and antenna 1320 are fixed to the machine and the transnntter position 
5 is readily determined by a matrix calculation as indicated above for any yaw, pitch and 
roll of the machine away from an initially calibrated orientation. The renderer then gets 
the operator head position and orientation in steps 1454 and 1456, by a determination of 
the position and orientation of the tracker receiver 1363 with respect to the tracker 
transmitter 1360, through the tracker processor 1360. A geometrical relationship 
1=0 between the tracker receiver and the operator's eyes is then assumed, such as described 
^2 in relation to FIG. 6a, to calculate the eye positions, and eventually the operator field of 
□ view. Information relating to the position, shape and attributes of virtual objects which 
^ are to be displayed is then obtained from database 420 in step 1460. Finally an image is 

created for each eye usirig the database information, and passed to the headset for 
15 display in step 1462. More detail for this last step has already been given in relation to 
^Cj FIG. 9 above. 

ju As noted, the Internet is a vast medium for both communication and storage of 

j^' information of many types. It would be desirable to make use of the Internet's 
information storage and commvinication potential in surveying and other related 

20 operations. Accordingly, an embodiment of the present invention is an augmented 

vision system which does so. This embodiment, which will now be described, is similar 
to the embodiments described above, but it includes a wireless hand-held 
communication device which enables the system to receive and use real-time updates of 
survey-related data for the user's current position from a remote server on the Internet 

25 (or other computer network), via a wireless telecommunications network. Some of the 
components of the augmented display system may also be connected to each other 
using a short-range wireless link, such as Bluetooth, infrared (IR) communication, or the 
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like. This approach enables the user to have a fully mteractive Internet experience with 
very little physical hardware. 

As is well known, many modern hand-held computing and communication 
devices are capable of accessing the Internet. For example, people can now browse the 
World Wide Web, send and receive email, instant messages, etc. using their cellular 
telephones, personal digital assistants (PDAs), and the like. The technology which 
makes this possible can be used to assist in surveying and related applications. For 
example, data relating to a job site can be maintained on a Web server on the Internet. 
This data can be continuously updated from any of a variety of sources. The updated 
data can then be transmitted via the Internet and a wireless telecommunications 
network to a user in the field, by way of the user's cellular telephone, PDA or other 
wireless device. The data received by the wireless communication device is then passed 
to an augmented vision system such as described above, to allow the user to view 
updated data, which may be of any of the types and formats discussed above. 

An embodiment of such a system is shown in FIG. 31. As shown, the system 
includes a wireless hand-held communication device 1605 (which may be a cellular 
telephone, PDA, or the like), a display processor 1606, and a headset 1607. The headset 
may be identical or similar to the headsets described above in connection with FIGS. 4b, 
5a and 5b. 

The communication device 1605 receives updated survey-related data associated 
with the user's current position from a remote Web server 1601 on the Internet 1602, via 
a wireless telecommunications network 1604. Note that in alternative embodiments, 
any of various other network types may be substituted for the Internet 1602 in FIG. 31, 
such as a corporate intranet, wide area network (WAN), or local area network (LAN). 
The data provided by the Web server 1601 may be in the form of virtual reality mark-up 
language (VRML) docimients, for example. The communication device may include a 
web browser (sometimes called a "minibrowser" or "microbrowser" when 
implemented in a hand-held device), using which the user can request data from the 
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Web server 1601. Alternatively, or in addition to this, the Web server 1601 might 
"push" data to the communication device 1605 with the data having been explicitly 
requested. 

As an alternative to VRML, the data transmitted by the Web server 1601 may be 
5 in a CAD (computer aided design) format. In that case, the browser of the 

communication device 1605 may include a 3-D "plug-in" to enable it to generate, from 
the received data, data suitable for displaying stereoscopic images. Alternatively, the 3- 
D functionality might instead be provided by the display processor 1606. 

The received data is used by the display processor 1606 to generate stereoscopic 
) images. The data provided by the Web server 1601 may include, for example, data on 
roads, points, lines, arcs, digital terrain models (DTMs), triangulated irregular network 
□ (TIN) models, or any of the other types of data discussed above. 

5 In the embodiment shown in FIG. 31, the wireless network 1604 is coupled to the 

1^ Internet 1602 by a gateway processing system ("gateway") 1603. The gateway 1603 

1 5 performs conventional functions for interconnecting two different types of networks, 

O 

'sj such as converting/translating between the protocols used by computers on the 
hi Internet, such as hypertext transport protocol (HTTP), and the protocols used by 
!^ communication devices on the wireless network 1607, such as wireless access protocol 
(WAP). 

20 Communication device 1605 may receive input from the user for operating the 

augmented vision system, such as to request updated data from Web server 1601, set 
preferences, etc. Display processor 1606 generates stereoscopic image data based on the 
received survey-related data and provides the image data to headset 1607 for display. 
As described above, headset 1607 has a substantially transparent display area to 

25 superimpose stereoscopic images of objects on a field of view of the user, based on the 
generated image data. If it is desired to display the objects as visually coregistered with 
real objects in the field of view, then the system will also include head orientation and 
eye position determining components such as discussed above. 
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As shown in FIG. 31, communication of data between communication device 
1605 and display processor 1606 may be via a short-range wireless link 1608, which may 
be a Bluetooth or IR link, for example. Alternatively, this connection may be a 
conventional wired link. Note that in other embodiments, the headset 1607 might also 
5 be connected to the display processor 1606 by a short-range wireless link such as any of 
the aforementioned, rather than a wired link. 

The display processor 1606 may be identical or similar to display processor 450 
described above (see FIG. 4b). Although display processor 1606 is shown as a separate 
device, in alternative embodiments it may be integrated with the headset 1607, with the 
tfi communication device 1605, or with a separate input device (if present). 
'2 In the illustrated embodiment, the commurucation device 1605 includes an input 

=P device in the form of a touchpad and/or various keys and buttons, which is sufficient to 
(3 allow the operator to control the functions of the system (requesting data, setting 

' preferences, etc.). In alternative embodiments, however, the system may include an 
16 input device that is separate from the communication device 1605, particularly if 
1= communication device 1605 has a very limited user interface. An example of such an 
Q embodiment is shown in FIG. 32. Accordingly, the embodiment of FIG. 32 includes a 
PDA or other separate input device 1610, separate from commimication device 1605, 
which is coupled to the commurucation device and /or the display processor 1611, by 
20 either a standard wired connection or a short-range wireless link, 1612. 

Rather than a touchpad and standard buttons /keys, the input device may 
alternatively be a virtual reaUty (VR) based device, such as a VR glove, pen, or wand, as 
described in connection with FIG. 19. The user could then interact with the system by 
pointing and tapping into the visual space. This approach, therefore, enables the user to 
25 have a fully interactive Internet experience with very little physical hardware. 

As noted above, the Web server 1601 may respond to requests from a web 
browser in the communication device 1605, or it may push data to the commimication 
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device 1605 independently of any request. The data provided by the Web server 1601 
to the augmented vision system may be received by the Web server 1601 from any of 
various sources, such as a design office 1614 or one or more roving data collectors 1616 
(e.g., surveyors, trucks, or dozers). The data may be in a proprietary format and/ or in a 
standard format (e.g., CAD). The data may be continuously and/or periodically 
updated on the Web server 1601 from these sources. The data may be loaded onto the 
Web server 1601 using any of various commimication chaimels, such as the Internet 
and/or a wireless network. Accordingly, the Web server 1601 may include algorithms 
to allow it to automatically aggregate the data, reduce or elinunate redundancies in the 
data and do any other appropriate data "clean-up", and generate VRML (or other 
similar) documents based on the data. Alternatively, these functions may be performed 
by human beings and /or other computer systems, such that the data is simply loaded 
into the Web server 1601 in a form ready to transmit to the user in the field. 

Note that if the volume of data to be provided from the Web server 1601 to the 
remote augmented vision system is very large, it may be impractical to download an 
entire file to the augmented vision system each time a minor update of the file is 
available. Consequently, it may be desirable to download only the changes to the data 
as the updates become available, such as by "streaming" the changes from the Web 
server 1601 to the augmented vision system. 

Thus, an augmented vision system for surveying and other applications has been 
described. Although the present invention has been described with reference to specific 
exemplary embodiments, it will be evident that various modifications and changes may 
be made to these embodiments without departing from the broader spirit and scope of 
the invention as set forth in the claims. Accordingly, the specification and drawings are 
to be regarded in an illustrative sense rather than a restrictive sense. 
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